Papers with Focus Attention
How Do Humans Write Code? Large Models Do It the Same Way Too (2024.emnlp-main)
Copied to clipboard
| Challenge: | Program-of-Thought (PoT) replaces natural language-based Chain-ofThough (CoT) but introduces more reasoning errors, such as incorrect formulas or flawed logic, compared to CoT. |
| Approach: | They propose a method that integrates CoT and Program-of-Thought to achieve more accurate reasoning and reinforcement learning. |
| Outcome: | The proposed method achieves an average improvement of 6.5% on the Llama-Base model and 4.3% on the Mistral-Bass model across 8 mathematical calculation datasets. |
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021.acl-long)
Copied to clipboard
| Challenge: | Currently, document summarization is challenging even for humans. |
| Approach: | They propose a focus attention mechanism which encourages decoders to generate tokens that are topically similar to the input document. |
| Outcome: | The proposed method outperforms top-k and nucleus sampling methods on the BBC extreme summarization task and is more accurate than focus attention-based models. |